Structure from Local Optima: Learning Subspace Juntas via Higher Order PCA

Independent Component Analysis (ICA), a well-known approach in statistics, assumes that data is generated by applying an affine transformation of a fully independent set of random variables, and aims to recover the orthogonal basis corresponding to the independent random variables. We consider a generalization of ICA, wherein the data is generated as an affine transformation applied to a product of distributions on two orthogonal subspaces, and the goal is to recover the two component subspaces. Our main result, extending the work of Frieze, Jerrum and Kannan, is an algorithm for generalized ICA that uses local optima of high moments and recovers the component subspaces. When one component is on a k-dimensional " relevant " subspace and satisfies some mild assumptions while the other is " noise " modeled as an (n − k)-dimensional Gaussian, the complexity of the algorithm is T (k, ǫ) + poly(n) where T depends only on the k-dimensional distribution. We apply this result to learning a k-subspace junta, i.e., an unknown 0-1 function in R n determined by an unknown k-dimensional subspace. This is a common generalization of learning a k-junta in R n and of learning an intersection of k halfspaces in R n , two important problems in learning theory. Our main tools are the use of local optima to recover global structure, a gradient-based algorithm for optimization over tensors, and an approximate polynomial identity test. Together, they significantly extend ICA and the class of k-dimensional labeling functions that can be learned efficiently.